Papers with evaluation task
Logic Haystacks: Probing LLMs’ Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding) (2026.eacl-short)
Copied to clipboard
| Challenge: | Recent large language models claim long context windows, but evaluations often involve simple retrieval tasks or synthetic tasks padded with irrelevant text. |
| Approach: | They use grammars to generate simplified English with logical representations to create long input text while controlling its semantics. |
| Outcome: | The proposed model performs better with realistic distractors than with standard models. |
Computer Assisted Annotation of Tension Development in TED Talks through Crowdsourcing (D19-59)
Copied to clipboard
| Challenge: | Using a neural network, we annotate whether tension is increasing, decreasing, or staying unchanged. |
| Approach: | They propose a machine-assisted method for the identification of tension development using a neural network based prediction model. |
| Outcome: | The proposed method is compared with other methods in in-house and crowdsourced environments. |
Neural Text Simplification of Clinical Letters with a Domain Specific Phrase Table (P19-1)
Copied to clipboard
| Challenge: | Clinical letters are written by doctors and typically contain complex medical language that is beyond the scope of the lay reader. |
| Approach: | They propose to augment existing neural text simplification software with a phrase table that links medical terminology to simpler vocabulary by mining SNOMED-CT data. |
| Outcome: | The proposed system is easier to understand than the existing system and the phrase table without it. |
A Closer Look at Linguistic Knowledge in Masked Language Models: The Case of Relative Clauses in American English (2020.coling-main)
Copied to clipboard
Marius Mosbach, Stefania Degaetano-Ortlieb, Marie-Pauline Krielke, Badr M. Abdullah, Dietrich Klakow
| Challenge: | Despite the high performance of transformer-based language models, we still lack understanding of the kind of linguistic knowledge they learn and rely on. |
| Approach: | They evaluate three transformer-based language models and test their grammatical and semantic knowledge by sentence-level probing, diagnostic cases, and masked prediction tasks. |
| Outcome: | The models capture grammatical and semantic knowledge, but they lack model-specific weaknesses especially on semantic knowledge. |
A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains (P19-1)
Copied to clipboard
| Challenge: | Existing models for diachronic and synchronic detection of lexical semantic divergences are superficial and lack of comparison. |
| Approach: | They propose to extend benchmark models on a common state-of-the-art evaluation task . they also demonstrate that the same evaluation task and modelling approaches can be utilised for synchronic detection of domain-specific sense divergences in the field of term extraction. |
| Outcome: | The proposed model can be utilised for the detection of domain-specific sense divergences in the field of term extraction. |
Word Embedding-Based Automatic MT Evaluation Metric using Word Position Information (N19-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics for machine translation are difficult to address word meaning because it is a surface-level metric. |
| Approach: | They propose to use word embeddings, sentence-level tf-idf, and cosine similarity between two word embeds as features, weight, and the distance between two features as features. |
| Outcome: | The proposed metric can evaluate machine translation based on word meaning . it achieves highest correlation with human judgment among several representative metrics. |
XNLI: Evaluating Cross-lingual Sentence Representations (D18-1)
Copied to clipboard
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, Veselin Stoyanov
| Challenge: | State-of-the-art natural language processing systems rely on annotated data to learn competent models. |
| Approach: | They extend the development and test sets of the Multi-Genre Natural Language Inference Corpus to 14 languages, including Swahili and Urdu. |
| Outcome: | The proposed evaluation set extends the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 14 languages including low-resource languages such as Swahili and Urdu. |
Automated Evaluation of Out-of-Context Errors (L18-1)
Copied to clipboard
| Challenge: | Existing methods to modify text understanding systems use only one sentence at a time . however, considering a larger context can improve performance for text understanding tasks. |
| Approach: | They propose to modify existing text data to insert out-of-context errors . they use a 2016 TEDTalk corpus to evaluate computational models for text understanding . |
| Outcome: | The proposed method targets real-world problems of transcription and translation systems by inserting authentic out-of-context errors. |
Recursive Routing Networks: Learning to Compose Modules for Language Understanding (N19-1)
Copied to clipboard
Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, Sandhini Agarwal, Joshua D. Greene, Dan Jurafsky, Christopher Potts, Lauri Karttunen
| Challenge: | Recursive Routing Networks are modular, adaptable models that learn effectively in diverse environments. |
| Approach: | They propose to apply Recursive Routing Networks (RRNs) to natural language understanding by integrating them into existing architectures and recurrent network hidden layers. |
| Outcome: | The proposed model optimizes the parameters of the functions and the meta-learner decision-making component for routing inputs through those functions. |
Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can be used to evaluate long documents, but they are limited by context window limitations. |
| Approach: | They propose to use granularity-aligned prompting and Focus Sentence Prompting to improve evaluation. |
| Outcome: | a new study shows that long texts lead to fewer error spans and reduced system ranking accuracy. |
Are Embedded Potatoes Still Vegetables? On the Limitations of WordNet Embeddings for Lexical Semantics (2023.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge Base Embedding (KBE) models are widely used to encode structured information from knowledge bases, including WordNet, but the evaluation task is often focused on link prediction, ignoring their semantic capabilities. |
| Approach: | They propose to evaluate the performance of Knowledge Base Embedding (KBE) models of WordNet on link prediction and their ability to encode semantic information. |
| Outcome: | The proposed model performs poorly on two semantic tasks and two downstream tasks. |
Revisiting Automated Topic Model Evaluation with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Topic models are an unsupervised dimensionality reduction technique that help organize large text collections. |
| Approach: | They propose to use large language models to evaluate document output and determine optimal number of topics. |
| Outcome: | The proposed model performs better on coherence ratings of word sets than on intrustion detection. |